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Isolating  the  Neural  Mechanisms  of  Interference  during 
Continuous  Multisensory  Dual-task  Performance 


Ryan  W.  Kasper1,  Hubert  Cecotti1,  Jon  Touryan2,  Miguel  P.  Eckstein1, 

and  Barry  Giesbrecht1 


Abstract 

■  The  need  to  engage  in  multiple  tasks  simultaneously  is  often 
encountered  in  everyday  experience,  but  coordinating  between 
two  or  more  tasks  can  lead  to  impaired  performance.  Typical 
investigations  of  multitasking  impairments  have  focused  on 
the  performance  of  two  tasks  presented  in  close  temporal 
proximity  on  discrete  trials;  however,  such  paradigms  do  not 
match  well  with  the  continuous  performance  situations  more 
typically  encountered  outside  the  laboratory.  As  a  result,  the 
stages  of  information  processing  that  are  affected  during  multi¬ 
sensory  continuous  dual  tasks  and  how  these  changes  in  pro¬ 
cessing  relate  to  behavior  remain  unclear.  To  address  these 
issues,  participants  were  presented  simultaneous  rapid  visual 
and  auditory  stimulus  sequences  under  three  conditions:  attend 
visual  only,  attend  auditory  only,  and  dual  attention  (attend 
both  visual  and  auditory).  Performance,  measured  in  terms  of 


INTRODUCTION 

Many  everyday  and  job-related  skills  require  continuous 
performance  of  multiple  tasks  that  involve  multiple  sen¬ 
sory  modalities.  For  example,  air-traffic  controllers  must 
simultaneously  and  continuously  monitor  both  auditory 
communications  from  pilots  and  visual  radar  displays. 
Drivers  may  choose  to  hold  a  cell  phone  conversation 
while  visually  scanning  the  road  for  other  vehicles.  It 
has  been  well  established  that  performing  multiple 
tasks  simultaneously  can  result  in  reduced  accuracy  and 
slower  RT  compared  with  when  those  tasks  are  com¬ 
pleted  in  isolation  (for  a  review,  see  Pashler,  1994). 
Dual-task  interference  has  classically  been  studied  using 
brief  punctate  trial-based  tasks,  such  as  in  studies  inves¬ 
tigating  the  psychological  refractory  period  (PRP)  in 
which  an  RT  increase  is  seen  when  the  delay  between 
two  discrete  tasks  is  reduced  (Telford,  1931).  This  RT 
increase  is  typically  interpreted  as  revealing  a  central 
resource  limitation  or  bottleneck  at  decision  or  response 
selection  stages  of  information  processing  (Zylberberg, 
Slezak,  Roelfsema,  Dehaene,  &  Sigman,  2010;  Lien, 
Ruthruff,  &  Johnston,  2006;  Sigman  &  Dehaene,  2005; 
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response  time  and  perceptual  sensitivity  (ct),  revealed  dual-task 
impairments  only  in  the  auditory  task.  Neural  activity,  measured 
by  the  ERP  technique,  revealed  that  both  early  stage  sensory 
processing  and  later  cognitive  processing  of  the  auditory  task 
were  affected  by  dual-task  performance,  but  similar  stages  of 
processing  of  the  visual  task  were  not.  Critically,  individual 
differences  in  neural  activity  at  both  early  and  late  stages  of  in¬ 
formation  processing  accurately  rank-ordered  individuals  based 
on  the  observed  difference  in  behavioral  performance  between 
the  single  and  dual  attention  conditions.  These  results  reveal 
relationships  between  behavioral  performance  and  the  neural 
correlates  of  both  early  and  late  stage  information  processing 
that  provide  key  insights  into  the  complex  interplay  between 
the  brain  and  behavior  when  multiple  tasks  are  performed 
continuously.  ■ 


Pashler,  1994;  de  Jong,  1993;  Welford,  1967,  but  for  an 
alternative  account,  see  Meyer  &  Kieras,  1997). 

Although  studies  of  the  PRP  phenomenon  give  insight 
to  the  cognitive  limitations  of  performing  two  successive 
tasks,  the  paradigm  only  imposes  dual-task  conditions  in 
short,  discrete  moments  in  time.  Thus,  a  distinction  can 
be  made  between  discrete  dual  tasks  that  are  performed 
in  short  overlapping  or  adjacent  periods,  such  as  the  PRP, 
and  continuous  dual  tasks  in  which  two  tasks  are  per¬ 
formed  simultaneously  for  extended  periods.  In  contrast 
to  the  limitations  observed  in  discrete  PRP  tasks,  studies 
of  continuous  dual  tasks,  such  as  simultaneously  moni¬ 
toring  two  visual  streams  for  targets,  have  reported  that 
RT  and  accuracy  often  do  not  exhibit  interference  effects 
relative  to  single  task  continuous  performance  (Allport, 
Antonis,  &  Reynolds,  1972).  The  lack  of  interference 
contrasts  the  effects  observed  in  trial-based  paradigms 
and  may  be  the  result  of  appropriate  task  scheduling 
and  switching  (Pashler,  1994).  Nevertheless,  there  are 
conditions  in  which  continuous  dual  tasks  can  result  in 
clear  performance  detriments,  such  as  impaired  driving 
while  talking  on  a  mobile  phone  (Just,  Keller,  &  Cynkar, 
2008;  Strayer,  Drews,  &  Crouch,  2006;  Strayer  &  Johnston, 
2001).  Auditory  tasks,  such  as  word  generation  or  listening 
to  passages  that  must  be  committed  to  memory,  can  cause 
performance  detriments  in  simultaneous  visual  tasks 
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(Gherri  &  Eimer,  2011;  Drews,  Pasupathi,  &  Strayer,  2008) 
and  result  in  reduced  visual  processing  (Uchiyama  et  al., 
2012).  However,  dual-task  interference  can  be  reduced, 
or  eliminated,  if  the  auditory  task  is  less  attentionally 
demanding  (Drews  et  al.,  2008;  Kunar,  Carter,  Cohen,  & 
Horowitz,  2008;  Strayer  &  Johnston,  2001).  Altogether, 
continuous  performance  of  dual  tasks  does  not  always 
result  in  performance  impairment,  but  shared  attentional 
resources  or  central  processing  bottlenecks  appear  to  be 
responsible  for  detriments  in  visual  performance  when 
auditory  information  is  simultaneously  attended  and  acted 
upon. 

A  complete  understanding  of  the  neural  mechanisms 
of  continuous  dual-task  performance  requires  under¬ 
standing  the  stage  of  information  processing  that  is  im¬ 
paired.  To  gain  insight  into  this  issue,  a  variety  of  studies 
have  recorded  EEG  data  during  the  performance  of  dis¬ 
crete  and  continuous  dual  tasks  (Kok,  1997).  For  example, 
EEG  studies  of  concurrent  visual  and  auditory  stimulation 
have  revealed  interference  of  neural  processing  in  visual 
tasks  (Jacoby,  Hall,  &  Mattingley,  2012;  Joassin,  Maurage, 
Bruyer,  Crommelinck,  &  Campanella,  2004;  Shams, 
Kamitani,  Thompson,  &  Shimojo,  2001).  ERPs  extracted 
from  continuous  EEG  data  have  also  been  used  to  isolate 
the  stage  of  neural  information  processing  that  is  affected 
by  dual-task  performance.  For  instance,  studies  using 
discrete  tasks  such  as  the  attentional  blink  task,  in  which 
a  performance  impairment  occurs  when  two  targets  are 
to  be  processed  within  a  few  hundred  milliseconds  of 
each  other  (Shapiro,  Raymond,  &  Arnell,  1997),  and  PRP 
tasks  have  found  that  early  visual  ERP  components  (e.g., 
PI,  Nl)  are  typically  unaffected  by  dual- task  performance 
(Marti,  Sigman,  &  Dehaene,  2012;  Sigman  &  Dehaene, 
2008;  Brisson,  Robitaille,  &Jolicoeur,  2007;  Sergent,  Baillet, 
&  Dehaene,  2005;  Luck,  1998;  Vogel,  Luck,  &  Shapiro,  1998). 
Investigations  of  visual-evoked  ERPs  in  continuous  dual 
tasks  have  been  less  common,  although  there  is  evidence 
that  attending  to  and  remembering  auditory  messages 
does  not  affect  the  visual  PI  component  evoked  during 
visual  search  tasks  (Gherri  &  Eimer,  2011). 

A  number  of  studies  have  investigated  the  modulation 
of  early  sensory  processing  by  measuring  auditory  ERPs 
in  continuous  dual- task  performance.  Of  particular  rele¬ 
vance  is  the  early  auditory  Nl  component,  which  is  sen¬ 
sitive  to  selective  spatial  attention  (Woldorff  et  al.,  1993; 
Hillyard,  Hink,  Schwent,  &  Picton,  1973).  The  amplitude 
of  the  auditory  Nl  is  known  to  scale  directly  with  the 
amount  of  priority  given  to  the  auditory  domain  in  con¬ 
tinuous  audiovisual  dual  tasks  at  fast  presentation  rates 
(Parasuraman,  1985).  Additionally,  the  continuous  perfor¬ 
mance  of  a  primary  visual  task  concurrently  with  a  sec¬ 
ondary  tone  monitoring  task  results  in  a  reduced  Nl 
amplitude  to  auditory  targets  compared  with  when  the 
auditory  monitoring  task  is  performed  in  isolation,  sug¬ 
gesting  that  the  Nl  can  index  the  allocation  of  attention 
to  auditory  stimuli  under  multimodal  dual-task  condi¬ 
tions  (Singhal,  Doerfling,  &  Fowler,  2002). 


Decision  and  response  selection  stages  of  information 
processing  during  dual-task  performance  have  also  been 
investigated  using  later  ERP  components.  The  most  fre¬ 
quently  studied  component  in  this  regard  is  the  P3,  a  posi¬ 
tive  deflection  starting  about  300-msec  poststimulus 
that  increases  in  strength  when  task-relevant  infrequent 
auditory  or  visual  stimuli  are  attended  and  detected 
(Duncan-Johnson  &  Donchin,  1977;  Sutton,  Braren,  Zubin, 
&  John,  1965).  Discrete  dual-task  studies  (e.g.,  using  the 
PRP  task)  have  found  delays  in  the  latency  of  the  visual 
P3,  supporting  the  idea  of  a  central  bottleneck  or  resource 
limitation  at  later  processing  stages  (Sigman  &  Dehaene, 
2008;  Dell’Acqua,  Jolicoeur,  Vespignani,  &  Toffanin,  2005; 
Arnell,  Helion,  Hurdelbrink,  &  Pasieka,  2004;  Luck,  1998). 
The  amplitude  of  the  visual  P3  is  known  to  be  greater  when 
a  visual  item  is  attended  in  isolation  compared  with  when 
attention  is  divided  (Mangun  &  Hillyard,  1990),  an  effect 
that  also  occurs  when  auditory  information  is  attended 
and  processed  simultaneously  with  a  visual  task  (Gherri 
&  Eimer,  2011).  Interestingly,  the  amplitude  of  the  P3  in 
divided  attention  tasks  is  inversely  related  to  the  difficulty 
of  the  primary  task  but  not  the  secondary  task  (Isreal, 
Chesney,  Wickens,  &  Donchin,  1980),  and  difficult  visual 
tasks  reduce  P3  amplitude  evoked  by  auditory  tasks 
(Wester,  Bocker,  Volkerts,  Verster,  &  Kenemans,  2008). 
Furthermore,  auditory  and  visual  P3  amplitudes  track  con¬ 
sistently  with  the  graded  amount  of  attention  devoted  to 
either  simultaneous  auditory  or  visual  tasks  (de  Jong, 
Toffanin,  &  Harbers,  2010),  which  is  consistent  with  a 
central  processing  limitation. 

When  considered  together,  the  extant  ERP  evidence 
suggests  that  dual-task  performance  typically  does  not 
modulate  the  earliest  stages  of  visual  information  pro¬ 
cessing  (i.e.,  <250  msec  after  stimulus  presentation)  but 
may  alter  early  stages  of  auditory  processing.  However, 
later  stages  of  information  processing  (i.e.,  >250  msec 
after  stimulus  presentation)  that  are  more  tightly  linked 
with  stimulus  categorization  and  recognition  in  both 
modalities  can  be  attenuated  and  delayed. 

This  Study 

Although  the  current  understanding  of  the  neural  mech¬ 
anisms  of  dual-task  performance  has  been  advanced  by 
ERP  studies,  there  remain  open  questions  about  the 
effects  of  continuous  dual-task  performance  on  both 
auditory  and  visual  information  processing.  Two  ques¬ 
tions  are  most  relevant  for  this  study.  First,  although  a 
number  of  studies  have  investigated  the  effects  of  con¬ 
tinuous  performance  dual  tasks  on  neural  processing 
using  ERPs,  these  have  tended  to  focus  on  later  stages 
of  processing  in  either  the  visual  or  auditory  domain. 
Second,  although  there  is  some  evidence  of  relationships 
between  neural  activity  measured  with  ERPs  and  behav¬ 
ioral  measures  in  single  tasks  (e.g.,  Das,  Giesbrecht,  & 
Eckstein,  2010;  Gerson,  Parra,  &  Sajda,  2005;  Kammer, 
Lehr,  &  Kirschfeld,  1999;  Mangun  &  Hillyard,  1990;  Kutas, 
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McCarthy,  &  Donchin,  1977),  the  effects  of  continuous 
dual  tasks  on  behavioral  performance  and  neural  activity 
measured  with  ERPs  have  typically  been  analyzed  sepa¬ 
rately  and  have  not  been  systematically  or  quantitatively 
related.  As  a  result,  it  remains  unclear  how  modulations 
of  visual  and  auditory  neural  processing  are  interrelated 
during  continuous  dual  tasks  and  how  these  modulations 
are  linked  to  changes  in  behavior. 

The  goal  of  this  study  was  to  address  these  issues  by 
measuring  changes  in  neural  processing  between  single 
and  dual-task  conditions,  in  both  visual  and  auditory 
modalities,  and  relating  these  directly  to  the  correspond¬ 
ing  changes  in  behavioral  task  performance.  EEG  was  re¬ 
corded  during  an  auditory  task  and  a  visual  task  that  each 
required  the  detection  of  targets  among  a  continuous 
stream  of  distractors.  These  tasks  were  performed  under 
three  attention  conditions:  attend  visual  only,  attend 
auditory  only,  and  attend  both  visual  and  auditory  (i.e., 
attend  dual).  Importantly,  the  auditory  and  visual  stimu¬ 
lation  was  identical  in  each  condition,  and  the  only  differ¬ 
ence  between  conditions  was  the  attentional  instruction. 
This  design  allowed  us  to  investigate  and  compare  the 
neural  correlates  of  continuous  auditory  and  visual 
dual-task  performance  on  behavior  and  different  stages 
of  information  processing  indexed  with  ERPs. 

METHODS 

Participants 

Sixteen  healthy  participants  (10  women)  ranging  in  age 
from  18  to  44  years  (mean  =  21.5  years,  SD  =  5-8  years) 
were  recruited  through  the  University  of  California,  Santa 
Barbara,  on-line  recruitment  system  and  received  either 
$20  an  hour  or  course  credit  for  participating.  All  vol¬ 
unteers  had  normal  or  corrected-to-normal  vision  and 


provided  informed  consent  before  the  experiment.  The 
University  of  California,  Santa  Barbara,  Human  Subjects 
Committee  approved  all  procedures. 

Stimuli  and  Materials 

Visual  stimuli  consisted  of  8-bit  grayscale  images  of  faces 
and  cars  taken  from  the  Max  Planck  Institute  for  Biologi¬ 
cal  Cybernetics  face  database  (Troje  &  Biilthoff,  1996). 
There  were  12  unique  images  of  faces  and  12  unique 
images  of  cars  (half  frontal  view,  half  45°  angle),  each 
filtered  to  match  the  average  power  spectrum  of  all 
images.  Independent  white  Gaussian  noise  fields  were 
filtered  by  this  average  power  spectrum  and  added  to 
the  original  24  face/car  images  in  10  iterations,  resulting 
in  240  images  (Das  et  al.,  2010).  Visual  stimuli  were  pre¬ 
sented  on  a  19-in.  ViewSonic  E90F  CRT  monitor  with  a 
resolution  of  1024  x  768  pixels  and  a  refresh  rate  of  75  Hz. 
Participants  sat  125  cm  from  the  monitor  in  a  darkened 
radio-frequency  shielded  chamber.  Images  subtended  a 
visual  angle  of  approximately  4.57°. 

Auditory  stimuli  consisted  of  words  from  the  military 
alphabet  recorded  using  a  computerized  text-to-speech 
function.  Ten  two-syllable  auditory  stimuli  were  used 
(bravo,  charlie,  delta,  echo,  kilo,  oscar,  tango,  victor,  xray, 
and  yankee),  with  the  average  duration  of  playback 
equaling  499  msec  and  a  standard  deviation  of  70  msec. 
Auditory  stimuli  were  presented  through  two  Dell  speakers 
placed  to  the  left  and  right  of  the  monitor. 

Procedure 

A  schematic  of  the  stimulus  sequence  is  displayed  in  Fig¬ 
ure  1.  Images  were  presented  at  fixation  at  a  rate  of  2  Hz 
(duration  =  500  msec,  ISI  =  0  msec).  The  visual  task  was 
designed  to  be  a  target  search  for  a  rare  item,  so  the 


Figure  1.  The  stimulus 
presentation  sequence  for 
the  visual  and  auditory  tasks. 
Stimulus  presentation  was 
identical  across  the  attend 
visual,  attend  auditory,  and 
dual  attention  conditions. 
For  display  purposes,  the 
noise  on  the  images  shown 
in  the  figure  is  less  than 
in  the  actual  experiment. 

For  further  details  on  the 
stimulus  presentation, 
see  Methods. 
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Figure  2.  The  behavioral 
results  are  displayed. 

(A)  Sensitivity  ( d' ')  of  the  target 
detection  responses  for  the 
visual  and  auditory  tasks.  (B)  RT 
of  the  target  detection  response 
for  the  visual  and  auditory  tasks. 
The  blue  lines  represent  the 
behavioral  performance  under 
single-task  attention  conditions 
(i.e.,  either  attend  visual  for  the 
visual  task  or  attend  auditory  for 
the  auditory  task),  and  the  red 
lines  represent  behavioral 
performance  under  the  dual 
attention  condition.  Error  bars 
represent  SEM. 


probability  of  a  face  image  was  held  constant  at  .10.  Audi¬ 
tory  stimuli  were  presented  at  a  slower  rate  (every  six 
images),  and  each  word  occurred  with  .10  probability. 
The  audiovisual  sequence  ran  uninterrupted  for  2-min 
intervals,  each  followed  by  a  self-paced  break.  To  avoid 
eye  blink  artifacts  in  the  EEG  data,  a  1-sec  blink  break 
was  offered  every  11  sec  during  the  stimulus  stream  by 
presenting  a  blank  screen  with  the  word  “Blink”  at  fixa¬ 
tion.  Participants  were  instructed  to  hold  their  blinks 
until  this  blink  break. 

Each  participant  performed  the  task  under  three  atten¬ 
tion  conditions.  In  the  attend  visual  condition,  partici¬ 
pants  were  instructed  to  monitor  the  images  for  the 
appearance  of  a  face  and  to  press  the  spacebar  key  as 
soon  as  a  face  image  was  detected.  Participants  were  in¬ 
structed  to  ignore  the  auditory  stimuli.  In  the  attend 
auditory  condition,  participants  were  instructed  to  moni¬ 
tor  the  auditory  stream  for  a  target  word  and  press  the 
return  key  when  the  target  was  recognized.  The  target 
word  was  randomly  selected  at  the  outset  of  the  auditory- 
only  task  and  was  held  constant  throughout  the  task 
condition.  During  the  attend  auditory  condition,  partici¬ 
pants  were  additionally  instructed  to  fixate  the  visual 
sequence  but  not  to  respond  to  any  of  the  images.  Finally, 
the  third  condition  was  the  attend  dual  condition  in  which 
participants  were  instructed  to  simultaneously  monitor 
the  visual  and  auditory  streams  for  targets.  The  target  word 
in  the  dual  task  was  selected  randomly  and  separately 
from  the  target  selection  in  the  auditory-only  task.  Partici¬ 
pants  were  instructed  to  give  the  auditory  and  visual  tasks 
equal  priority  in  the  dual  attention  condition.  The  order 
of  the  conditions  was  randomized  and  counterbalanced 
across  participants. 


It  is  important  to  note  that  the  visual  and  auditory  stim¬ 
uli  were  identical  in  all  conditions  of  the  task,  with  the 
only  difference  being  instructions  to  focus  on  one  modal¬ 
ity  over  another.  A  total  of  4,800  images  (4,320  cars, 
480  faces)  and  800  auditory  words  (720  nontarget  words, 
80  target  words)  were  presented  in  each  condition. 


Behavioral  Data  Analysis 

Behavioral  performance  was  measured  using  RT  and  sen¬ 
sitivity  (</).  RTs  were  computed  as  the  time  between  the 
previous  stimulus  presentation  and  the  button  press. 
These  values  were  averaged  across  trials  for  each  partici¬ 
pant  and  condition.  To  index  the  level  of  stimulus  deci¬ 
sion  accuracy  in  each  task,  the  signal  detection  theory 
sensitivity  measure  of  d!  was  used  (Macmillan  &  Creelman, 
1991;  Green  &  Swets,  1966).  First,  the  hit  rate  (HR)  was 
calculated  by  the  number  of  correct  target-present  re¬ 
sponses  divided  by  the  total  number  of  targets.  Second, 
the  false  alarm  rate  (FA)  was  calculated  as  the  number  of 
target-present  responses  given  to  nontargets  relative  to 
the  total  number  of  nontargets.  The  d!  was  then  calculated 
by  subtracting  the  2r  transformations:  d!  =  z(HR)  —  2r(FA). 
The  resulting  d!  measure  represents  the  individual  par¬ 
ticipant’s  sensitivity  in  discriminating  between  signal  (i.e., 
target)  and  noise  (i.e.,  nontarget)  stimuli.  To  test  for  dif¬ 
ferences  between  conditions  for  behavioral  measures,  a 
2  (Task  modality)  X  2  (Attention  condition  —  single  and 
dual  conditions)  repeated-measures  AN OVA  was  used  for 
both  RT  and  d! .  We  also  report  the  mean  (M),  SEM, 
repeated- measures  t  statistic  (t),  probability  (p),  and  effect 
size  (r|2)  for  any  direct  comparisons  (Figure  2). 
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EEG  Data  Acquisition 

The  EEG  was  measured  for  each  participant  using  a 
BioSemi  ActiveTwo  system  consisting  of  32  Ag-AgCl  sin¬ 
tered  electrodes  mounted  in  an  elastic  cap  and  placed 
according  to  the  International  10-20  system.  Additional 
electrodes  were  placed  at  the  right  and  left  mastoids, 
as  well  as  1  cm  lateral  to  the  left  and  right  external 
canthi  (horizontal),  and  above  and  below  each  eye 
(vertical)  for  the  EOG.  Data  were  sampled  at  512  Hz 
and  referenced  offline  to  the  average  mastoid  signal. 
Trials  in  which  EOG  channels  showed  a  change  in 
amplitude  of  ±75  microvolts  or  greater  were  classified 
as  eye  blink/movements  and  were  excluded  from  further 
analysis. 

ERP  Data  Analysis 

To  test  whether  dual  attention  conditions  affected  spe¬ 
cific  ERP  components,  mean  amplitude  and  peak  latency 
analyses  were  performed.  We  chose  to  focus  specifically 
on  the  early  visual  PI  and  auditory  N1  components  to  index 
changes  in  early  perceptual  processing.  Although  the 
short  stimulus  presentation  (500  msec)  in  this  study  makes 


it  difficult  to  completely  capture  the  full  P3  component, 
we  do  report  additional  analyses  of  amplitude  and  latency 
for  visual  and  auditory  P3  to  index  later  processing 
changes.  Components  were  identified  by  first  finding  the 
peak  across  participants  and  conditions  within  classically 
defined  time  windows  and  electrodes  (e.g.,  100-150  msec 
at  occipital  sites  for  the  PI)  and  centering  a  40-msec  time 
window  on  that  peak  time  point.  The  resulting  windows 
that  were  used  were  91-131  msec  for  the  PI  and  126- 
166  msec  for  the  Nl.  A  larger  window  of  400-600  msec 
was  used  for  the  P3  component  because  of  its  more 
extended  and  coarse  waveform  (Mangun  &  Hillyard, 
1990)  and  was  cut  off  at  600  msec  to  avoid  confounds  from 
the  stimulus  response  of  the  next  trial.  Plots  of  the  ampli¬ 
tude  topography  were  then  produced  for  each  time 
window  and  condition  (see  Figure  3),  and  the  subset  of 
contiguous  electrodes  showing  the  strongest  amplitude 
for  that  component,  independent  of  stimulus  condition, 
was  selected  by  visual  inspection  and  used  to  create  the 
averaged  ERP  waveform.  This  selection  process  resulted 
in  the  following  sets  of  electrodes:  Ol,  02,  and  Oz  for 
the  visual  PI;  Cz,  CPI,  CP2,  and  Pz  for  the  visual  P3;  FC1, 
FC2,  Cz,  CPI,  and  CP2  for  the  auditory  Nl;  and  Cz,  CPI, 
CP2,  Pz,  P3,  and  P4  for  the  auditory  P3. 
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Figure  3-  Topographic  plots  of  ERP  amplitude.  (A)  The  mean  amplitudes  during  the  time  windows  for  the  visual  PI  (91-131  msec)  and  auditory  Nl 
(126-166  msec)  are  displayed  separately  for  target  and  nontarget  stimuli  within  each  attention  condition.  (B)  The  mean  amplitudes  during  the  time 
windows  for  the  visual  and  auditory  P3  (400-500  msec)  are  displayed  separately  for  target  and  nontarget  stimuli  within  each  attention  condition. 
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The  ERP  mean  amplitudes  were  calculated  by  aver¬ 
aging  the  amplitude  across  the  chosen  time  window  for 
the  electrode  set  for  each  component.  Latencies  for  the 
ERP  components  were  calculated  by  measuring  the  time 
in  milliseconds  from  stimulus  onset  to  the  peak  ampli¬ 
tude  within  the  component  time  window.  To  test  the  ef¬ 
fect  of  dual-task  attention  on  ERP  component  amplitudes 
and  latencies,  an  ANOVA  with  three  factors  for  attention 
condition  (attend  visual,  attend  auditory,  dual  attention) 
was  performed  for  each  component  separately  for  target 
and  nontarget  stimuli. 

Relating  ERPs  to  Behavioral  Performance 

To  determine  if  the  dual  attention  condition  caused  dif¬ 
ferences  in  the  ERP  components  that  were  related  to  cor¬ 
responding  dual  attention  differences  in  behavioral 
performance,  a  rank  order  analysis  (Kasper,  Elliott,  & 
Giesbrecht,  2012;  Das  et  al.,  2010)  was  performed  that 
is  more  robust  to  deviations  from  linear  relationships 
than  the  Pearson  correlation  coefficient.  This  rank  order 
analysis  considered  the  differences  in  ERP  and  behavioral 
measures  between  single  attention  (i.e.,  attend  visual  or 
attend  auditory)  and  dual  attention  conditions.  Rank 
order  analyses  were  performed  by  comparing  pairs  of 
participants  to  determine  whether  the  difference  in 
amplitude  or  latency  of  an  ERP  component  elicited  by 
targets  predicted  the  difference  in  behavior  for  each  pair. 
Thus,  rank  order  accuracy  was  assigned  a  value  of  1  if  an 
attenuation  in  ERP  amplitude  or  delay  in  peak  latency 
from  single  to  dual  attention  conditions  predicted  a 
reduction  in  d!  or  slowing  of  RT.  For  example,  if  par¬ 
ticipant  A  had  a  larger  PI  amplitude  reduction  (single 
attention  —  dual  attention)  than  participant  B,  and  partici¬ 
pant  A  also  had  a  larger  d'  reduction  (single  attention  — 
dual  attention),  that  rank  ordering  was  considered  accu¬ 
rate.  As  a  result,  rank  order  accuracies  above  chance 
(0.50)  indicate  a  positive  relationship  between  behavior 
and  ERP  changes,  whereas  accuracies  below  chance  indi¬ 
cate  a  negative  relationship.  The  rank  order  process  was 
done  for  all  possible  pairs  of  participants  to  create  an 
average  accuracy.  A  jackknife  method  that  left  out  one 
participant  for  each  full  cycle  of  comparisons  was  used 
to  compute  standard  error  (Duda,  Hart,  &  Stork,  2001). 
Tests  of  significance  were  done  using  an  independent 
measures  t  test  against  a  random  permutation  rank  accu¬ 
racy  (M  =  0.50,  SEM  =  0.03).  Because  of  the  possibility  of 
an  inflated  type  I  error  rate  resulting  from  multiple  com¬ 
parisons,  the  false  discovery  rate  (FDR)  correction  q  values 
are  reported  along  with  typical  p  values  (Benjamini  & 
Hochberg,  1995). 

RESULTS 

The  results  are  presented  in  four  separate  sections:  The 
first  section  contains  the  results  of  the  analysis  of  behav¬ 


ioral  performance,  the  second  section  contains  the  analysis 
of  early  stages  of  information  processing  (i.e.,  <250  msec 
after  stimulus  presentation),  the  third  section  contains 
the  analysis  of  later  stages  of  information  processing  (i.e., 
>250  msec  after  stimulus  presentation),  and  the  fourth 
section  contains  the  analysis  that  relates  the  dual-task  mod¬ 
ulations  in  neural  activity  measured  with  ERPs  and  changes 
in  behavioral  performance. 


Behavioral  Performance 

Mean  d!  and  RT  are  shown  as  a  function  of  task  and  atten¬ 
tion  condition  in  Figure  2.  Mean  d'  was  larger  in  the  audi¬ 
tory  task  when  detecting  words  (M  =  3-23,  SEM  =  0.10) 
compared  with  visual  task  when  detecting  faces  (M  = 

I. 96,  SEM  =  0.11;  F(  1,  15)  =  87.65,  p  <  .01,  T|2  = 
0.85).  No  other  main  effects  or  interactions  for  d!  were 
significant  (both  ^(l,  15)  <  2.36,  p  >  .14,  g2  <  0.14). 
RTs  were  faster  in  the  visual  task  ( M  =  309.96,  SEM  = 
7.32)  compared  with  the  auditory  task  (M  =  379.18, 
SEM  =  11.68;  F(l,  15)  =  32.12,^  <  .01,  q2  =  0.68).  Addi¬ 
tionally,  RTs  were  faster  under  the  single  attention  con¬ 
dition  (M  =  332.44,  SEM  =  7.50)  compared  with  the  dual 
attention  condition  (M  =  356.71,  SEM  =  8.42;  F(l,  15)  = 
23-93, p  <  -01,  rf  =  0.62).  These  effects  were  qualified  by 
an  interaction  between  task  modality  and  attention  condi¬ 
tions  (F(  1, 15)  =  16.91, p  <  .01,  r[2  =  0.53).  Post  hoc  t  tests 
revealed  that  this  interaction  was  driven  by  the  fact  that, 
on  the  one  hand,  the  auditory  task  RTs  were  faster  in 
the  attend  auditory  condition  (M  =  357.66,  SEM  = 

II. 63)  compared  with  the  dual  attention  condition  (M  = 

400.71,  SEM  =  13-23;  £(15)  =  4.88,  p  <  .01),  whereas  on 
the  other  hand,  there  was  no  difference  between  attend 
visual  (M  =  307.22,  SEM  =  7.44)  and  dual  attention  (M  = 

312.71,  SEM  =  7.64)  conditions  in  the  visual  task  (£(  15)  = 
1.52, p  >  .15). 


Measuring  Early  Stages  of  Information  Processing 

To  understand  the  effects  of  continuous  dual-task  per¬ 
formance  on  early  stages  of  information  processing,  we 
isolated  the  visual  PI  and  auditory  N1  ERP  components. 
Comparisons  were  made  for  component  amplitude  and 
latency  between  the  three  different  attention  conditions 
(attend  visual,  attend  auditory,  dual  attention)  separately 
for  the  target  and  nontarget  stimuli.  These  comparisons 
allow  us  to  address  the  effects  of  attention  on  the  neural 
processing  of  visual  and  auditory  information.  Table  1 
shows  a  summary  of  the  important  findings  for  each  of 
the  comparisons  made. 

Visual  Target  Evoked  Responses 

The  mean  ERP  waveforms  evoked  by  the  visual  stimuli 
are  shown  in  Figure  4A.  The  mean  amplitude  for  targets 
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Table  1.  ERP  Effects  in  the  Continuous  Dual  Task 


Temporal  Processing 
Stage 

ERP 

Component 

Stimulus 

Type 

ERP 

Measure 

Key  Result 

Early 

Visual  PI 

Target 

Amplitude 

Reduced  in  attend  auditory  vs. 
dual  attention  condition 

Latency 

Delayed  in  attend  auditory  condition 

Nontarget 

Amplitude 

No  differences 

Latency 

Delayed  in  attend  auditory  condition 

Auditory  N1 

Target 

Amplitude 

Enhanced  in  attend  auditory  condition1 

Latency 

No  differences 

Nontarget 

Amplitude 

Enhanced  in  attend  auditory  vs.  attend 
visual  condition1 

Latency 

Delayed  in  attend  visual  vs. 
attend  auditory  condition 

Late 

Visual  P3 

Target 

Amplitude 

Reduced  in  attend  auditory  condition 

Latency 

No  differences 

Nontarget 

Amplitude 

No  differences 

Latency 

No  differences 

Auditory  P3 

Target 

Amplitude 

Reduced  in  attend  visual  condition 

Latency 

Delayed  in  dual  vs.  auditory, 
dual  vs.  visual,  and  auditory  vs. 
visual  attention  conditions1 

Nontarget 

Amplitude 

No  differences 

Latency 

No  differences 

Table  1  shows  a  summary  of  the  relevant  results  from  the  ERP  component  analyses. 
“Those  comparisons  in  which  there  was  an  effect  of  dual-task  performance. 


differed  significandy  as  a  function  of  attention  condition 
(F( 2,  30)  =  3-7 9,p  <  .04,  rf  =  0.20).  Post  hoc  comparisons 
revealed  that  this  interaction  was  driven  by  smaller  over¬ 
all  PI  amplitude  in  the  attend  auditory  condition  (M  = 
3.29,  SEM  =  0.49)  compared  with  the  dual  attention 
condition  (M  =  3-98,  SEM  =  0.58;  £(15)  =  2.70,  p  <  .02). 
Direct  comparison  between  the  attend  visual  (M  =  3-74, 
SEM  =  0.59)  and  dual- task  conditions  indicated  that  the 
visual  PI  did  not  differ  significantly  (£(15)  =  1.63,  p  > 
.10).  Similar  to  the  mean  amplitude  analyses,  the  analysis 
of  the  PI  peak  latencies  to  targets  also  revealed  an  effect 
of  attention  (F( 2,  30)  =  7.24,  p  <  .01,  T|2  =  0.33),  which 
resulted  from  a  later  PI  peak  in  the  attend  auditory  (M  = 
118.26,  SEM  =  1.60)  compared  with  both  the  attend 
visual  (M  =  115.94,  SEM  =  1.30)  and  dual  attention 
(M  =  115-58,  SEM  =  1.60)  conditions  (both  £s(15)  > 
2.69, ps  <  .02). 

Visual  Nontarget  Evoked  Responses 

The  amplitude  analysis  of  the  PI  evoked  by  the  non  targets 
revealed  no  effect  of  attention  (F(2,  30)  =  3-29,  p  >  .05, 


rf  =  0.18).  Similar  to  the  analysis  of  the  latencies  of  the 
PI  evoked  by  visual  targets,  there  was  a  significant  effect 
of  attention  on  the  latency  of  the  PI  evoked  by  non  targets 
(F(2,  30)  =  8.59,  p  <  .01,  rf  =  0.36).  Direct  comparisons 
revealed  that  the  PI  peaked  later  in  the  attend  auditory 
(M  =  118. 51,  SEM  =  1.52)  than  both  the  attend  visual  (M  = 
115.70,  SEM  =  1.29)  and  dual  (M  =  115.82,  SEM  =  1.33) 
conditions  (both  £s (15)  >  2.97,  ps  <  .01). 

Auditory’  Target  Evoked  Responses 

Figure  4B  shows  the  average  ERPs  evoked  by  the  auditory 
stimuli.  Mean  amplitude  analyses  of  the  auditory  N1  to 
target  stimuli  revealed  a  significant  effect  of  attention 
(F( 2,  30)  =  9.73, p  <  .01,  rf  =  0.39).  Direct  comparisons 
revealed  an  enhanced  (i.e.,  more  negative)  N1  in  the  attend 
auditory  condition  (M  =  —4.89,  SEM  =  0.65)  compared 
with  both  the  dual  attention  (M  =  —2.58,  SEM  =  0.69) 
and  attend  visual  (M  =  —2.65,  SEM  =  0.55)  conditions 
(both  £s(15)  >  4.00,  ps  <  .01).  There  was  no  effect  of 
attention  condition  on  the  target  N1  latency  (F( 2,  30)  = 
2 .64, p  >  .09,  rf  =  0.15). 


482  Journal  of  Cognitive  Neuroscience 


Volume  26,  Number  3 


11 


Auditory  Nontarget  Evoked  Responses 

The  effect  of  attention  on  N1  amplitude  elicited  by  tar¬ 
gets  was  mirrored  in  the  analysis  of  the  N1  amplitude 
to  nontarget  stimuli  (F( 2,  30)  =  4.28,  p  <  .03,  ip  = 
0.22).  Direct  comparisons  revealed  that  the  attend  audi¬ 
tory  condition  (M  =  —3-64,  SEM  =  0.47)  had  significantly 
larger  N1  amplitude  compared  with  the  attend  visual 
condition  (M  =  —2.84,  SEM  =  0.45;  £(  15)  >  2.67,  p  < 
.02).  There  was  a  trend  for  a  larger  N1  in  the  attend 
visual  condition  compared  with  the  dual  task  (M  = 
—3-07,  SEM  =  0.40;  £(15)  =  1.86,  p  >  .08).  There  was 
a  significant  effect  of  attention  on  non  target  N1  latency 
(F(2,  30)  =  4.13, p  <  -03,  rf  =  0.22).  Direct  comparisons 
indicated  a  significantly  later  N1  in  the  attend  visual 


Figure  4.  Average  ERP 
waveforms  for  visual  and 
auditory  stimuli.  (A)  The  visual 
PI  waveforms  are  displayed 
separately  for  target  stimuli  on 
the  left  and  nontarget  stimuli 
on  the  right.  (B)  The  auditory 
N1  waveforms  are  displayed 
separately  for  target  stimuli  on 
the  left  and  nontarget  stimuli 
on  the  right.  (C)  The  visual 
P3  waveforms  are  displayed 
separately  for  target  stimuli  on 
the  left  and  nontarget  stimuli 
on  the  right.  (D)  The  auditory 
P3  waveforms  are  displayed 
separately  for  target  stimuli 
on  the  left  and  nontarget 
stimuli  on  the  right.  Blue  lines 
represent  the  attend  visual 
condition,  green  lines  represent 
the  attend  auditory  condition, 
and  red  lines  represent  the 
dual  attention  condition. 

The  gray  bar  represents  the 
time  window  used  for  mean 
amplitude  and  peak  latency 
analyses  for  the  visual  PI 
(91-131  msec),  auditory  N1 
(126-166  msec),  and  the 
visual  and  auditory  P3 
components  (400-600  msec). 


condition  (M  =  138.40,  SEM  =  2.77)  compared  with 
the  attend  auditory  condition  (M  =  145.61,  SEM  = 
3.25;  £(15)  =  2.19,  p  <  .05).  There  was  a  trend  toward 
a  later  N1  peak  latency  in  the  attend  visual  compared 
with  the  dual  attention  condition  (M  =  145.00,  SEM  = 
3.09;  £(15)  =  2.04,  p  >  .06). 

Measuring  Late  Stages  of  Information  Processing 

To  understand  the  effects  of  continuous  dual-task  per¬ 
formance  on  the  later  stages  of  information  processing, 
we  isolated  the  P3  ERP  component  for  visual  and  audi¬ 
tory  stimuli.  A  summary  of  these  results  is  presented  in 
Table  1. 
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Figure  5.  Rank  order  analysis 
using  the  amplitude  difference 
(attend  visual/auditory  —  dual 
attention)  to  order  the 
behavioral  outcome  changes 
of  participant  pairs.  (A)  The 
average  rank  order  accuracy 
using  amplitude  measures 
from  the  visual  PI,  auditory  Nl, 
visual  P3,  and  auditory  P3  ERP 
components.  (B)  The  average 
rank  order  accuracy  using 
latency  measures  from  the 
visual  PI,  auditory  Nl, 
visual  P3,  and  auditory  P3 
ERP  components.  Blue  bars 
represent  the  rank  order 
accuracy  of  changes  in 
sensitivity  d'  measures, 
and  red  bars  represent  the 

rank  order  accuracy  of  changes  in  RT  measures.  Error  bars  represent  SEM  calculated  using  a  jackknifing  method.  The  dashed  lines  reflect  the 
95%  confidence  interval  for  a  random  permutation  test.  Single  asterisks  (*)  represent  significance  at  p  <  .05,  and  double  asterisks  (**)  represent 
significance  at  p  <  .01. 


Visual  Target  Evoked  Responses 

Figure  4C  shows  the  average  P3  ERP  waveforms.  There 
was  a  significant  effect  of  attention  on  the  visual  P3  elic¬ 
ited  by  target  face  images  (F( 2,  30)  =  25-35, p  <  .01,  if  = 
0.63).  Direct  comparisons  revealed  that  this  effect  was 
driven  by  a  reduced  P3  amplitude  in  the  attend  auditory 
condition  (M  =  0.09,  SEM  =  0.38)  in  comparison  with 
the  attend  visual  (M  =  4.66,  SEM  =  0.66)  and  dual  atten¬ 
tion  (M  =  3-97,  SEM  =  0.66)  conditions  (both  ts(15)  > 
5-37,  ps  <  .01).  There  was  no  significant  attention  effect 
on  P3  latency  (F( 2,  30)  =  1.74,  p  >  .19,  if  =  0.10). 


Visual  Nontarget  Evoked  Responses 

There  was  no  effect  of  attention  on  P3  amplitude  elic¬ 
ited  by  nontarget  stimuli  (F( 2,  30)  =  0.82,  p  >  .45,  if  = 
0.05).  There  was  also  no  effect  of  attention  on  the  P3 
latency  to  nontarget  stimuli  (F( 2,  30)  =  0.72,  p  >  .49, 
if  =  0.05). 


Auditory  Target  Evoked  Responses 

The  grand-averaged  ERPs  for  the  auditory  P3  component 
are  shown  in  Figure  4D.  The  audit ory-evoked  P3  ampli¬ 
tude  to  targets  showed  a  significant  effect  of  attention 
(F(. 2,  30)  =  16.34,  p  <  .01,  if  =  0.52).  Direct  compari¬ 
sons  indicated  that  the  P3  in  the  attend  auditory  condi¬ 
tion  (M  =  4.98,  SEM  =  1.04)  and  the  dual- task  condition 
(M  =  3-41,  SEM  =  0.60)  were  both  larger  than  in  the 
attend  visual  condition  (M  =  -1.27,  SEM  =  0.95;  both 
£s(15)  >  4.46,  ps  <  .01),  although  the  attend  auditory 
and  dual-task  conditions  were  not  significantly  different 
from  each  other  (t(\  5)  =  1.48,  p  >  .15).  For  P3  latency 
to  auditory  targets,  there  was  a  significant  effect  of  atten¬ 


tion  (F(2,  30)  =  8.16, )D  <  .01,  if  =  0.35).  Direct  compar¬ 
isons  revealed  that  the  P3  in  the  dual-task  condition  (M  = 
582.98,  SEM  =  23.13)  peaked  significantly  later  than  in 
both  the  attend  auditory  (M  =  555.52,  SEM  =  39.51) 
and  attend  visual  (M  =  517.55,  SEM  =  72.62)  conditions 
(both  £s(15)  >  2.73,  ps  <  .02).  Furthermore,  the  peak  in 
the  attend  visual  condition  was  earlier  than  in  the  attend 
auditory  condition  (£(15)  =  2.14,  p  <  .05). 

Auditory  Nontarget  Evoked  Responses 

There  was  no  effect  of  attention  condition  for  nontarget 
P3  amplitude  (F( 2,  30)  =  0.60, p  >  .55,  if  =  0.04).  There 
was  also  no  effect  of  attention  for  P3  latency  to  non¬ 
targets  (F(2,  30)  =  0.79,  p  >  .46,  if  =  0.05). 


Measuring  the  Relationship  between  Behavioral 
Performance  and  Neural  Activity 

To  investigate  the  relationship  between  neural  activity 
and  behavior,  we  performed  several  rank  order  ana¬ 
lyses.  These  analyses  were  done  separately  using  dif¬ 
ferences  in  ERP  amplitude  and  ERP  latency  from  single 
to  dual  attention  conditions  (i.e.,  attend  dual  —  attend 
visual/auditory). 

Amplitude  Differences 

Even  in  the  absence  of  dual  attention  differences  at  the 
group  level,  such  as  with  the  visual  PI  amplitude,  it  is 
plausible  that  individual  variability  in  the  ERP  amplitudes 
would  be  effective  at  rank-ordering  individuals  in  terms 
of  their  behavioral  performance.  The  drop-off  in  ampli¬ 
tude  of  the  visual  PI  to  targets  was  not  effective  at  rank 
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ordering  individuals  in  terms  of  their  behavioral  perfor¬ 
mance  measured  with  either  d'  (M  =  0.46,  SEM  = 
0.03)  or  RT  (M  =  0.47,  SEM  =  0.03;  both  £s(30)  <  1.08, 
p s  >  .10,  qs  >  0.20).  However,  the  reduction  in  the 
amplitude  of  the  N1  in  the  dual  attention  condition  rela¬ 
tive  to  the  attend  auditory  condition  could  successfully 
rank  order  the  corresponding  dual-task  changes  in  d' 
(M  =  0.70,  SEM  =  0.03;  £(30)  =  7.89,  p  <  .01,  q  < 
0.01)  but  not  the  changes  in  RT  (M  =  0.51,  SEM  = 
0.03;  £( 30)  =  0.35,  p  >  -73,  q  >  0.73).  These  results 
indicate  that  the  reductions  in  performance  in  the  dual¬ 
task  condition  relative  to  the  single-task  condition  mea¬ 
sured  with  d'  are  associated  with  attenuated  early  sensory 
processing  measured  by  the  auditory  N1  evoked  by 
targets. 

For  the  visual-evoked  P3,  reductions  in  amplitude 
were  significantly  related  to  both  declines  in  visual  task 
d!  (M  =  0.68,  SEM  =  0.03)  and  delays  in  RT  (M  =  0.63, 
SEM  =  0.02;  both  fs(30)  >  5-90,  ps  <  .01,  qs  <  0.01). 
Changes  in  the  auditory  P3  were  similarly  related  to 
delays  in  RT  (M  =  0.58,  SEM  =  0.03;  £(30)  =  2.93,  p  < 
.01,  q  <  0.02).  However,  in  contrast  to  the  visual  P3,  reduc¬ 
tions  in  auditory  P3  amplitude  were  negatively  related  to 
d!  performance  ( M  =  0.43,  SEM  =  0.03;  £(30)  =  2.60, p  < 
.02,  q  <  0.03).  Consequently,  it  appears  that  reductions 
in  the  visual  P3  amplitude  were  related  to  d'  and  RT 
decrements,  whereas  reductions  in  auditory  P3  ampli¬ 
tude  were  related  to  improved  d'  but  slower  RT.  Fig¬ 
ure  5A  shows  the  results  of  these  amplitude  rank  order 
analyses. 

Latency  Differences 

Although  there  were  no  differences  between  single  and 
dual  attention  conditions  in  latency  for  visual  PI  or  audi¬ 
tory  N1  components  elicited  by  targets,  individual  differ¬ 
ences  in  latency  may  still  be  related  to  differences  in 
behavior.  The  change  in  latency  of  the  visual  PI  did  sig¬ 
nificantly  rank  order  changes  in  d'  above  chance  (M  = 
0.56,  SEM  =  0.02;  £(30)  =  2.48,  p  <  .02,  q  <  0.04)  but 
did  not  significantly  rank  order  changes  in  RT  (M  = 
0.53,  SEM  =  0.02;  £(30)  =  1.55,  p  >  .13,  q  >  0.17).  Dif¬ 
ferences  in  the  auditory  N1  latency  did  not  significantly 
rank  order  d!  (M  =  0.49,  SEM  =  0.03;  £(30)  =  0.41,  p  > 
.68,  q  >  0.68)  but  did  significantly  rank  order  RT  differ¬ 
ences  (M  =  0.73,  SEM  =  0.02;  £(30)  =  9.75 ,p  <  .01,  q  < 
0.01).  In  other  words,  delayed  visual  PI  latencies  from 
single  to  dual  attention  conditions  were  related  to  de¬ 
creases  in  d! ,  whereas  delayed  auditory  N1  latencies  were 
related  to  delayed  RTs. 

Changes  in  the  latency  of  the  P3  component  from  the 
visual  task  were  not  significantly  related  to  changes  in 
either  d'  (M  =  0.50,  SEM  =  0.02)  or  RT  (M  =  0.46, 
SEM  =  0.02;  both  £s(30)  <  1.65,  ps  >  .11,  qs  >  0.15). 
Changes  in  the  auditory  P3  latency  were  significantly 
related  to  both  decreases  in  d'  (M  =  0.62,  SEM  = 
0.02)  as  well  as  delays  in  RT  (M  =  0.58,  SEM  =  0.02; 


both  £s(30)  >  3-79,  ps  <  .01,  qs  <  0.01).  These  results 
are  displayed  in  Figure  5B. 

DISCUSSION 

This  study  was  designed  to  investigate  the  effects  of 
multisensory  continuous  dual-task  performance  on  the 
early  and  late  stages  of  information  processing  by  mea¬ 
suring  visual-  and  auditory-evoked  neural  activity  using 
the  ERP  technique  and  relating  the  changes  in  neural 
activity  to  individual  differences  in  behavior.  The  results 
revealed  that  dual-task  impairments  in  information  pro¬ 
cessing  were  mainly  restricted  to  the  auditory  task, 
with  both  early  and  late  auditory  processing  stages 
being  affected  by  the  dual  attention  condition.  Further¬ 
more,  individual  differences  in  both  early  and  late 
ERP  components  were  related  to  individual  differences 
in  behavioral  performance.  These  results  provide  key 
insights  to  the  neural  mechanisms  underlying  the  perfor¬ 
mance  of  concurrent  continuous  tasks  in  multiple  sen¬ 
sory  modalities. 

Dual-task  Modulations  of  Behavioral  Performance 

There  were  two  noteworthy  patterns  of  behavioral  per¬ 
formance.  First,  across  all  conditions,  the  visual  task 
was  performed  less  accurately  than  the  auditory  task. 
Behavioral  responses  in  the  visual  task  were  also  faster 
compared  with  the  auditory  task,  suggesting  a  gener¬ 
alized  speed-accuracy  trade-off.  Second,  the  level  of 
performance  on  the  visual  task  was  unaffected  by  the 
simultaneous  demands  of  the  auditory  task.  In  contrast, 
the  auditory  task  showed  reduced  RT  and  an  unchanged 
d!  when  the  visual  task  was  performed  concurrently  in 
the  dual  attention  condition.  The  lack  of  a  dual-task 
performance  decrement  on  visual  task  discrimination 
sensitivity  is  consistent  with  studies  showing  that  audi¬ 
tory  tasks  do  not  always  interfere  with  visual  task  perfor¬ 
mance  (Drews  et  al.,  2008;  Kunar  et  al.,  2008). 

There  are  two  differences  between  the  visual  and 
auditory  tasks  that  may  account  for  the  behavioral  re¬ 
sults.  First,  the  stimulus  presentation  rate  was  faster 
in  the  visual  task  than  in  the  auditory  task.  Second, 
the  stimuli  in  the  visual  task  were  embedded  in  noise 
and  briefly  presented,  whereas  the  auditory  stimuli 
were  not  degraded.  The  combination  of  these  differ¬ 
ences  likely  resulted  in  the  auditory  task  being  less  dif¬ 
ficult  than  the  visual  task,  a  proposal  that  is  supported 
by  the  higher  mean  d!  in  the  auditory  task.  The  high 
level  of  difficulty  of  the  visual  task  compared  with  the 
auditory  task,  a  result  of  increased  presentation  rate 
and  degraded  stimuli,  may  have  led  participants  to 
place  priority  on  this  task  over  the  auditory  task, 
although  they  were  instructed  to  give  the  tasks  equal 
priority.  Indeed,  the  increased  RT  in  the  auditory  task 
during  the  dual  attention  condition  suggests  that  par¬ 
ticipants  were  sacrificing  performance  in  the  easier 
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auditory  task  to  maintain  performance  levels  in  the  visual 
task. 

Dual-task  Modulations  of  Neural  Activity 

If  participants  were  prioritizing  the  visual  task  because  of 
its  difficulty,  then  the  dual  attention  condition  should 
affect  auditory  processing  but  not  visual  processing. 
Consistent  with  this  hypothesis,  the  amplitude  and 
latency  of  the  visual  PI  component  were  the  same  in 
the  attend  visual  and  attend  dual  conditions.  Although 
the  PI  amplitude  was  similar  between  single  and  dual 
tasks  for  both  targets  and  nontargets,  visual  stimuli  in 
the  attend  auditory  condition  elicited  a  smaller  amplitude 
and  later  peaking  PI  component.  This  finding  is  in  line 
with  previous  studies  showing  that  the  PI  is  enhanced 
when  attended  compared  with  when  ignored  (Hillyard, 
Teder-Salejarvi,  &  Munte,  1998;  Mangun  &  Hillyard, 
1990).  Interestingly,  despite  the  lack  of  group  differences 
between  the  attend  visual  and  dual  attention  conditions 
in  terms  of  the  PI  latency  to  targets,  rank  order  analyses 
revealed  that  participants  who  had  more  delayed  PI 
peaks  in  the  dual  attention  versus  attend  visual  condi¬ 
tion  also  tended  to  have  lower  d!  in  the  dual  attention 
compared  with  the  attend  visual  condition.  This  may 
mean  that  the  concurrent  performance  of  both  the  visual 
and  auditory  tasks  delayed  early  visual  processing  and 
the  degree  of  this  delay  contributed  to  reductions  in 
the  ability  to  accurately  detect  targets  in  the  visual  task. 

Consistent  with  the  hypothesis  that  performance  in 
the  auditory  task  was  sacrificed  in  favor  of  the  visual  task 
in  the  dual  attention  condition,  the  amplitude  of  the 
auditory  N1  was  smaller  in  response  to  target  stimuli  in 
the  attend  dual  condition  compared  with  the  attend  audi¬ 
tory  condition.  The  influence  of  the  dual  task  on  the 
amplitude  of  the  auditory  N1  evoked  by  targets  indicates 
that  early  processing  of  auditory  stimuli  was  disrupted 
when  attention  was  split  between  the  visual  and  auditory 
tasks,  a  finding  consistent  with  previous  studies  (e.g., 
Singhal  et  al.,  2002).  Similarly,  RT  in  the  auditory  task 
was  increased  in  the  dual  task  compared  with  the  single 
task.  Unlike  previous  studies,  however,  we  investigated 
how  these  changes  in  ERP  amplitudes  related  to  the 
changes  in  behavioral  performance.  The  smaller  N1 
amplitude  caused  by  the  dual  attention  condition  was 
strongly  related  to  the  corresponding  decrements  in  d! 
for  each  participant.  Furthermore,  delays  in  the  N1  peak 
induced  by  the  dual  attention  condition  were  strongly 
related  to  delays  in  RT.  These  findings  suggest  that  the 
variations  in  N1  amplitude  and  latency  to  auditory  targets 
under  dual  attention  conditions  can  be  used  to  accurately 
index  and  even  predict  the  level  of  auditory  processing 
interference  in  a  dual  task. 

Similar  to  the  PI  amplitude  analysis,  the  amplitude  of 
the  visual  target  evoked  P3  was  larger  in  attend  visual  and 
dual  tasks  compared  with  the  attend  auditory  condition. 
This  is  consistent  with  previous  studies  showing  larger  P3 


amplitudes  when  infrequent  stimuli  are  attended  and 
detected  (Duncan-Johnson  &  Donchin,  1977;  Sutton  et  al., 
1965).  However,  other  studies  have  found  reduced  P3 
amplitudes  when  attention  is  divided  (de  Jong  et  al., 
2010;  Mangun  &  Hillyard,  1990),  which  does  not  match 
with  the  similar  P3  amplitudes  evoked  by  visual  targets 
under  single-  and  dual-task  conditions  observed  here.  This 
disparity  could  be  explained  by  (a)  the  hypothesized  in¬ 
creased  priority  given  to  the  visual  task  over  the  auditory 
task,  (b)  the  finding  that  the  P3  for  a  primary  task  can 
be  unaffected  by  changes  in  difficulty  of  a  secondary  task 
(Singhal  et  al.,  2002),  (c)  the  participants  engaged  in  a 
strategy  of  switching  between  tasks  because  of  the  fixed 
presentation  rate  in  the  auditory  task,  or  (d)  some  com¬ 
bination  thereof.  Nevertheless,  the  delay  in  auditory  P3 
to  targets  during  the  dual  task  is  consistent  with  delays 
attributed  to  a  processing  bottleneck  in  discrete  dual-task 
studies  (Sigman  &  Dehaene,  2008;  Dell’Acqua  et  al.,  2005; 
Arnell  et  al.,  2004;  Luck,  1998).  When  considered  along 
with  the  amplitude  reduction  seen  in  the  dual-task  N1  to 
targets,  this  provides  further  support  for  the  idea  that 
participants  in  this  experiment  sacrificed  auditory  task  per¬ 
formance  to  prioritize  the  visual  task  under  dual  attention 
conditions. 

The  rank  order  relationships  between  changes  in 
behavioral  performance  and  P3  amplitude  and  latency 
changes  indicate  that  later  neural  processing  stages  are 
affected  by  dual-task  conditions  in  a  way  that  is  also  re¬ 
lated  to  behavior.  These  results  match  well  with  previous 
findings  that  P3-related  ERP  activity  elicited  by  visual 
stimuli  could  explain  large  amounts  of  variance  in  RT 
(Gerson  et  al.,  2005).  Although  the  negative  relationship 
between  auditory  P3  amplitude  and  reduced  d'  was  un¬ 
expected,  this  effect  may  reflect  individual  differences 
in  strategic  speed-accuracy  trade-offs.  Because  the  same 
auditory  P3  amplitude  reductions  are  related  to  de¬ 
creases  in  RT,  smaller  P3  responses  in  the  dual  task 
appear  to  have  led  to  slower  RT  but  either  no  reductions 
or  modest  increases  in  d' .  This  potential  trade-off  effect  is 
small  compared  with  the  other  P3  rank  order  effects. 
Moreover,  the  latency  delays  in  the  same  auditory  P3 
components  are  very  strongly  related  to  both  decreases 
in  d'  and  delays  in  RT.  Similar  to  the  Nl,  changes  in  the 
visual  and  auditory  P3  could  be  used  to  index  changes  in 
behavior  in  dual-task  environments,  such  as  is  currently 
done  with  brain-computer  interfaces  (Cecotti,  Kasper, 
Elliott,  Eckstein,  &  Giesbrecht,  2011;  Gevins  &  Smith, 
2003). 

Implications  and  Conclusions 

The  finding  that  overall  visual  processing  was  unaffected 
by  the  simultaneous  attention  to  the  auditory  word 
monitoring  task  supports  the  idea  that  auditory  process¬ 
ing  does  not  necessitate  visual  interference  (Gherri  & 
Eimer,  2011;  Kunar  et  al.,  2008).  Instead,  previous  evi¬ 
dence  shows  that,  whereas  simply  listening  to  auditory 
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information  may  not  interfere  with  visual  processing 
(Drews  et  al.,  2008),  producing  auditory  messages  (Kunar 
et  al.,  2008)  and  actively  storing  auditory  messages  in 
memory  (Gherri  &  Eirner,  2011)  can  cause  interference 
with  visual  tasks  measured  by  both  behavior  and  neural 
activity.  The  present  findings  further  reveal  that  pro¬ 
cessing  of  auditory  stimuli,  here,  single  words  that  must 
be  identified  and  compared  with  the  known  target 
word,  may  not  be  sufficient  to  interfere  with  visual  pro¬ 
cessing.  On  the  other  hand,  the  present  results  seem  to 
clearly  indicate  visual  task  interference  on  auditory 
information  processing  at  both  early  and  late  stages. 
The  precise  nature  of  this  interference  is  unclear  based 
on  the  present  experiment;  however,  the  pattern  of 
results  is  consistent  with  the  notion  that,  at  some  stage 
of  processing,  vision  and  audition  may  share  a  single 
pool  of  attentional  resources  in  continuous  dual-task 
situations  (Arrighi,  Lunardi,  &  Burr,  2011)  or  that  both 
modalities  require  access  to  a  single  capacity  limited 
mechanism  that  is  subject  to  information  processing 
bottlenecks  (Sigman  &  Dehaene,  2008;  DelTAcqua 
et  al.,  2005;  Arnell  et  al.,  2004;  Luck,  1998;  Pashler, 
1994).  Importantly,  within  the  present  context,  the  inter¬ 
action  across  modalities  is  likely  partially  driven  by  the 
difficulty  of  the  visual  task,  which  may  have  served  to 
either  draw  extra  resources  that  would  otherwise  have 
been  used  for  the  auditory  task  or  to  create  an  informa¬ 
tion  processing  bottleneck. 

What  is  clear  from  the  present  data  is  that  the  re¬ 
source  sharing  or  bottleneck  responsible  for  the 
changes  in  auditory-evoked  responses  are  first  occur¬ 
ring  within  200  msec  of  stimulus  presentation,  suggest¬ 
ing  that  the  information  that  is  being  affected  is 
relatively  early  in  the  information  processing  stream. 
The  N 1  has  previously  been  localized  to  auditory  cortex 
(Woldorff  et  al.,  1993),  and  the  dual-task  modulation  of 
this  N 1  response  likely  reflects  attenuated  early  sensory 
processing  under  dual-task  conditions.  On  the  other 
hand,  the  P3  modulations  around  400-600  msec  after 
stimulus  presentation,  which  is  thought  to  reflect  changes 
in  attention  at  later  decision  stages  in  higher-order  brain 
regions  (Mangun  &  Hillyard,  1990;  Duncan-Johnson  & 
Donchin,  1977),  may  be  mediated  by  a  separate  resource 
pool  or  bottleneck.  Consistent  with  this  notion,  a  separate 
control  analysis  did  not  reveal  a  correlation  between  dual¬ 
task  modulations  in  N1  and  P3  amplitude  or  latency  (all 
r  values  <  0.17,  p  >  .55).  Although  we  cannot  determine 
whether  the  observed  patterns  of  data  reflect  resource 
sharing  or  a  bottleneck,  the  present  data  demonstrate 
separate  dual-task  detriments  in  both  early  and  later  audi¬ 
tory  neural  processing. 

It  is  difficult  to  fully  dissociate  the  sensory  modality 
and  the  difficulty  of  the  tasks  in  this  study,  and  as  a  result, 
the  differences  arising  between  the  visual  and  auditory 
tasks  cannot  be  attributed  to  a  single  cause  without 
further  experimentation.  Previous  work  has  shown  simi¬ 
lar  auditory  N1  and  P3  effects  with  systematically  manipu¬ 


lated  difficulty  in  audiovisual  dual  tasks  (de  Jong  et  al., 
2010;  Singhal  et  al.,  2002),  but  these  studies  used  dif¬ 
ferent  types  of  tasks  and  did  not  completely  characterize 
the  relationship  between  neural  activity  and  behavior. 
Future  work  using  continuous  performance  tasks  is 
needed  to  relate  neural  signals  from  both  modalities  to 
behavior  while  also  manipulating  the  difficulty  of  the 
two  tasks.  Furthermore,  additional  studies  should  con¬ 
tinue  to  emphasize  the  relationships  between  behavior 
and  properties  of  the  N1  and  P3,  as  these  relationships 
could  potentially  be  used  as  a  reliable  index  and  predic¬ 
tor  of  detection  performance. 

In  summary,  the  present  results  support  the  con¬ 
clusion  that,  during  a  multisensory  continuous  perfor¬ 
mance  dual  task,  auditory  processing  was  reduced  by 
the  dual-task  demands  whereas  visual  processing  was 
unaffected,  a  pattern  that  may  have  resulted  from  strat¬ 
egies  to  mitigate  the  increased  difficulty  of  the  visual 
task.  Furthermore,  the  results  also  support  the  conclu¬ 
sion  that  the  increased  attentional  demands  imposed 
by  a  multisensory  continuous  performance  dual  task 
not  only  modulate  neural  activity  associated  with  both 
early  and  late  stages  of  information  processing  but  that 
these  modulations  are  tightly  linked  with  behavioral 
performance. 
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